Three-dimensional image pickup apparatus, light-transparent unit, image processing apparatus, and program

ABSTRACT

A 3D image capture device according to an embodiment includes: a light transmitting section  1  with a transmitting area  1   a , of which the spectral transmittance characteristic varies in a first direction; an image sensor  2   a  which is arranged to receive light that has been transmitted through the light transmitting section  1  and which outputs a photoelectrically converted signal representing the light received; and an image processing section which extracts an edge of a subject in the first direction, which is included in an image that has been generated based on the photoelectrically converted signal supplied from the image sensor  2   a , and which estimates information about the depth of the subject based on a lightness or hue pattern of a background in the vicinity of the edge extracted.

TECHNICAL FIELD

The present invention relates to a single-lens 3D image capturingtechnology for generating a parallax image using a single optical systemand a single image sensor.

BACKGROUND ART

Recently, the performance and functionality of digital cameras anddigital movie cameras that use some solid-state image sensor such as aCCD and a CMOS (which will be sometimes simply referred to herein as an“image sensor”) have been enhanced to an astonishing degree. Inparticular, the size of a pixel structure for use in a solid-state imagesensor has been further reduced these days thanks to rapid developmentof semiconductor device processing technologies, thus getting an evengreater number of pixels and drivers integrated together in asolid-state image sensor. As a result, the resolution of an image sensorhas lately increased rapidly from around one million pixels to tenmillion or more pixels in a matter of few years. On top of that, thequality of an image captured has also been improved significantly aswell. As for display devices, on the other hand, LCD and plasma displayswith a reduced depth now provide high-resolution and high-contrastimages, thus realizing high performance without taking up too muchspace. And such video quality improvement trends are now spreading from2D images to 3D images. In fact, 3D display devices that achieve highimage quality although they require the viewer to wear a pair ofpolarization glasses have been developed just recently.

As for the 3D image capturing technology, a typical 3D image capturedevice with a simple arrangement uses an image capturing system with twocameras to capture a right-eye image and a left-eye image. According tothe so-called “two-lens image capturing” technique, however, two camerasneed to be used, thus increasing not only the overall size of the imagecapture device but also the manufacturing cost as well. To overcome sucha problem, methods for capturing multiple images with parallax (whichwill be sometimes referred to herein as a “multi-viewpoint image”) byusing a single camera have been researched and developed. Such a methodis called a “single-lens image capturing method”.

For example, Patent Document No. 1 discloses a scheme that uses twopolarizers, of which the transmission axes cross each other at rightangles, and a rotating polarization filter. FIG. 14 is a schematicrepresentation illustrating an arrangement for an image capturing systemthat adopts such a scheme. This image capturing system includes a0-degree-polarization polarizer 11, a 90-degree-polarization polarizer12, a reflective mirror 13, a half mirror 14, a circular polarizationfilter 15, a driver 16 that rotates the circular polarization filter 15,an optical lens 3, and an image capture device 9 for capturing the imagethat has been produced by the optical lens. In this arrangement, thehalf mirror 14 reflects the light that has been transmitted through thepolarizer 11 and then reflected from the reflective mirror 13 buttransmits the light that has been transmitted through the polarizer 12.With such an arrangement, the light beams that have been transmittedthrough the two polarizers 11 and 12, which are arranged at twodifferent positions, pass through the half mirror 14, the circularpolarization filter 15 and the optical lens 3 and then enter the imagecapture device 9, where an image is captured. The image capturingprinciple of this scheme is that two images with parallax are capturedby rotating the circular polarization filter 15 so that the light beamsthat have been incident on the two polarizers 11 and 12 are imaged atmutually different times.

According to such a scheme, however, images at mutually differentpositions are captured time-sequentially by rotating the circularpolarization filter 15, and therefore, two images with parallax cannotbe captured at the same time, which is a problem. In addition, thedurability of such a system is also a question mark because the systemuses mechanical driving. On top of that, since the incoming light passesthrough the polarizers 11, 12 and the polarization filter 15, thequantity of the light received eventually by the image capture device 9decreases by as much as 50%, which is non-negligible, either.

To overcome these problems, Patent Document No. 2 discloses a scheme forcapturing two images with parallax at the same time without using suchmechanical driving. An image capture device that adopts such a schemegets the two incoming light beams, which have come from two differentdirections, condensed by a reflective mirror, and then received by animage sensor in which two different kinds of polarization filters arearranged alternately, thereby capturing two images with parallax withoutusing a mechanical driving section.

FIG. 15 is a schematic representation illustrating an arrangement for animage capturing system that adopts such a scheme. This image capturingsystem includes two polarizers 11 and 12, of which the transmission axesare arranged to cross each other at right angles, reflective mirrors 13,an optical lens 3, and an image sensor 2. On its image capturing plane,the image sensor 2 has a number of pixels 10 and polarization filters 17and 18, each of which is provided one to one for an associated one ofthe pixels 10. Those polarization filters 17 and 18 are arrangedalternately over all of those pixels. In this case, the transmissionaxis directions of the polarization filters 17 and 18 agree with thoseof the polarizers 11 and 12, respectively.

With such an arrangement, the incoming light beams are transmittedthrough the polarizers 11 and 12, reflected from the reflective mirrors13, passed through the optical lens 3 and then incident on the imagecapturing plane of the image sensor 1. Those light beams to betransmitted through the polarizers 11 and 12, respectively, and thenincident on the image sensor 1 are transmitted through the polarizationfilters 17 and 18 and then photoelectrically converted by the pixelsthat are located right under those polarization filters 17 and 18. Ifthe images to be produced by those light beams that have beentransmitted through the polarizers 11 and 12 and then incident on theimage sensor 1 are called a “right-eye image” and a “left-eye image”,respectively, then the right-eye image and the left-eye images aregenerated by a group of pixels that face the polarization filters 17 anda group of pixels that face the polarization filter 18, respectively.

As can be seen, according to the scheme disclosed in Patent Document No.2, two kinds of polarization filters, of which the transmission axes arearranged so as to cross each other at right angles, are arrangedalternately over the pixels of the image sensor, instead of using thecircular polarization filter disclosed in Patent Document No. 1. As aresult, although the resolution decreases to a half compared to themethod of Patent Document No. 1, a right-eye image and a left-eye imagewith parallax can be obtained at the same time by using a single imagesensor. According to such a technique, however, the incoming light hasits quantity decreased considerably when being transmitted through thepolarizers and the polarization filters, and therefore, the quantity ofthe light received by the image sensor decreases as significantly as inPatent Document No. 1.

To cope with such a problem of the decreased quantity of light received,Patent Document No. 3 discloses a technique for obtaining two imageswith parallax and a normal image with a single image sensor. Accordingto such a technique, those two images with parallax and the normal imagecan be obtained by a single image sensor by changing mechanically somecomponents that have been used to capture two images with parallax withalternative components for use to capture a normal image, and viceversa. When two images with parallax are going to be obtained, twopolarization filters are arranged on the optical path as disclosed inPatent Document No. 2. On the other hand, when a normal image is goingto be obtained, those polarization filters are mechanically removed fromthe optical path. By introducing such a mechanism, those images withparallax and a normal image that uses the incoming light highlyefficiently can be obtained.

Although a polarizer or a polarization filter is used according to thetechniques disclosed in Patent Document Nos. 1 to 3, color filters mayalso be used according to another approach. For example, Patent DocumentNo. 4 discloses a technique for obtaining two images with parallax atthe same time using color filters. FIG. 16 schematically illustrates animage capturing system that adopts such technique. The image capturingsystem that uses that technique includes a lens 3, a lens diaphragm 19,a light beam confining plate 20 with two color filters 20 a and 20 bthat have mutually different transmission wavelength ranges, and aphotosensitive film 21. In this case, the color filters 20 a and 20 bmay be filters that transmit red- and blue-based light rays,respectively.

In such an arrangement, the incoming light passes through the lens 3,the lens diaphragm 19 and the light beam confining plate 20 and producesan image on the photosensitive film. In the meantime, only red- andblue-based light rays are respectively transmitted through the two colorfilters 20 a and 20 b of the light beam confining plate 20. As a result,a magenta-based color image is produced on the photosensitive film bythe light rays that have been transmitted through the two color filters.In this case, since the color filters 20 a and 20 b are arranged atmutually different positions, the image produced on the photosensitivefilm comes to have parallax. Thus, if a photograph is developed with thephotosensitive film and viewed with a pair of glasses, in which red andblue films are attached to its right- and left-eye lenses, the viewercan view an image with depth. In this manner, according to the techniquedisclosed in Patent Document No. 4, a multi-viewpoint image can beproduced using the two color filters.

According to the technique disclosed in Patent Document No. 4, the lightrays are imaged on the photosensitive film, thereby producing imageswith parallax there. Meanwhile, Patent Document No. 5 discloses atechnique for producing images with parallax by transforming incominglight into electrical signals. FIG. 17 schematically illustrates a lightbeam confining plate according to Patent Document No. 5. According tosuch a technique, a light beam confining plate 22, which has a red raytransmitting R area 22R, a green ray transmitting G area 22G and a blueray transmitting B area 22B, is arranged on a plane that intersects withthe optical axis of the imaging optical system at right angles. And bygetting the light rays that have been transmitted through those areasreceived by a color image sensor that has red-, green- andblue-ray-receiving R, G and B pixels, an image is generated based on thelight rays that have been transmitted through those areas.

Patent Document No. 6 also discloses a technique for obtaining imageswith parallax using a similar configuration to the one used in PatentDocument No. 5. FIG. 18 schematically illustrates a light beam confiningplate as disclosed in Patent Document No. 6. According to thattechnique, by making the incoming light pass through R, G and B areas23R, 23G and 23B of the light beam confining plate 23, images withparallax can also be produced.

Patent Document No. 7 also discloses a technique for generating multipleimages with parallax using a pair of filters with mutually differentcolors, which are arranged symmetrically to each other with respect toan optical axis. By using red and blue filters as the pair of filters,an R pixel that senses a red ray observes the light that has beentransmitted through the red filter, while a B pixel that senses a blueray observes the light that has been transmitted through the bluefilter. Since the red and blue filters are arranged at two differentpositions, the light received by the R pixel and the light received bythe B pixel have come from mutually different directions. Consequently,the image observed by the R pixel and the image observed by the B pixelare ones viewed from two different viewpoints. And by definingcorresponding points between those images on a pixel-by-pixel basis, themagnitude of parallax can be calculated. And based on the magnitude ofparallax calculated and information about the focal length of thecamera, the distance from the camera to the subject can be obtained.

Patent Document No. 8 discloses a technique for obtaining informationabout a subject distance based on two images that have been generatedusing either a diaphragm to which two color filters with mutuallydifferent aperture sizes (e.g., red and blue color filters) are attachedor a diaphragm to which two color filters in two different colors areattached horizontally symmetrically with respect to the optical axis.According to such a technique, if light rays that have been transmittedthrough the red and blue color filters with mutually different aperturesizes are observed, the degrees of blur observed vary from one color toanother. That is why the degrees of blur of the two images that areassociated with the red and blue color filters vary according to thesubject distance. By defining corresponding points with respect to thoseimages and comparing their degrees of blur to each other, informationabout the distance from the camera to the subject can be obtained. Onthe other hand, if light rays that have been transmitted through twocolor filters in two different colors that are attached horizontallysymmetrically with respect to the optical axis are observed, thedirection from which the light observed has come changes from one colorto another. As a result, two images that are associated with the red andblue color filters become images with parallax. And by definingcorresponding points with respect to those images and calculating thedistance between those corresponding points, information about thedistance from the camera to the subject can be obtained.

According to the techniques disclosed in Patent Documents Nos. 4 to 8mentioned above, images with parallax can be produced by arranging RGBcolor filters on a light beam confining plate. However, since a lightbeam confining plate is used, the percentage of the incoming light thatcan be used decreases significantly. In addition, to increase the effectof parallax, those RGB color filters should be arranged at distantpositions and should have decreased areas. In that case, however, thepercentage of the incoming light that can be used further decreases.

Unlike these techniques, Patent Document No. 9 discloses a technique forobtaining multiple images with parallax and a normal image that is freefrom the light quantity problem by using a diaphragm in which RGB colorfilters are arranged. According to that technique, when the diaphragm isclosed, only the light rays that have been transmitted through the RGBcolor filters are received. On the other hand, when the diaphragm isopened, the RGB color filter areas are outside of the optical path, andtherefore, the incoming light can be received entirely. Consequently,images with parallax can be obtained when the diaphragm is closed and anormal image that uses the incoming light highly efficiently can beobtained when the diaphragm is opened.

CITATION LIST Patent Literature

-   Patent Document No. 1: Japanese Laid-Open Patent Publication No.    62-291292-   Patent Document No. 2: Japanese Laid-Open Patent Publication No.    62-217790-   Patent Document No. 3: Japanese Laid-Open Patent Publication No.    2001-016611-   Patent Document No. 4: Japanese Laid-Open Patent Publication No.    2-171737-   Patent Document No. 5: Japanese Laid-Open Patent Publication No.    2002-344999-   Patent Document No. 6: Japanese Laid-Open Patent Publication No.    2009-276294-   Patent Document No. 7: Japanese Laid-Open Patent Publication No.    2010-38788-   Patent Document No. 8: Japanese Laid-Open Patent Publication No.    2010-79298-   Patent Document No. 9: Japanese Laid-Open Patent Publication No.    2003-134533

Non-Patent Literature

-   Non-Patent Document No. 1: Yuta MORIUE, Takeshi TAKAKI, and Idaku    ISHII, A Real-time Monocular Stereo System Using a Viewpoint    Switching Iris, Transactions of the 27^(th) Annual Conference of the    Robotics Society of Japan, 3R2-06, 2009.

SUMMARY OF INVENTION Technical Problem

According to any of these techniques of the related art, amulti-viewpoint image can be certainly generated, but the quantity ofthe light received by the image sensor is smaller than usual because apolarizer or color filters are used. In order to receive a sufficientquantity of incoming light, some mechanism that removes the polarizingportion or color filter areas from the optical path needs to be used.That is to say, according to none of these techniques of the relatedart, a multi-viewpoint image and an image that uses the incoming lighthighly efficiently can be obtained at the same time without using such amechanism.

Also, according to the related art, in estimating depth informationbased on the multi-viewpoint image obtained, feature portions areextracted from the multiple images and matched to each other, therebyestimating the depth information. Alternatively, as disclosed in PatentDocument No. 6, the depth information can also be estimated bycalculating a pixel shift based on a linear color model in an RGB colorspace.

An embodiment of the present invention provides an image capturingtechnique by which both an image that uses incoming light highlyefficiently and depth information can be obtained at the same timewithout using the known depth information estimating method.

Solution to Problem

In order to overcome the problem described above, a 3D image capturedevice as an embodiment of the present invention includes: a lighttransmitting section with a transmitting area, of which the spectraltransmittance characteristic varies in a first direction; an imagesensor which is arranged to receive light that has been transmittedthrough the light transmitting section and which outputs aphotoelectrically converted signal representing the light received; animaging section which produces an image on the image capturing plane ofthe image sensor; and an image processing section which extracts an edgeof a subject in the first direction, which is included in an image thathas been generated based on the photoelectrically converted signalsupplied from the image sensor, and which estimates information aboutthe depth of the subject based on a lightness or hue pattern of abackground in the vicinity of the edge extracted.

This general and particular aspect of the present invention can begenerally implemented as a system, a method, a computer program or acombination thereof.

Advantageous Effects of Invention

A 3D image capture device as an embodiment of the present invention cantransform information about the depth of a subject into informationabout the brightness or color of an image, and therefore, can calculatedepth information. Also, in one embodiment, by increasing thetransmittance of the rest of the light transmitting section other thanthe transmitting area, depth information and a high-sensitivity imagecan be obtained at the same time.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 A block diagram illustrating an overall configuration for animage capture device as a first exemplary embodiment.

FIG. 2 Schematically illustrates a general arrangement of alight-transmitting plate, an optical lens and an image sensor in thefirst exemplary embodiment.

FIG. 3 A front view of the light-transmitting plate according to thefirst exemplary embodiment.

FIG. 4 Illustrates the basic color arrangement of the image capturingsection of a solid-state image sensor according to the first exemplaryembodiment.

FIG. 5 Schematically illustrates how an image is shot according to thefirst exemplary embodiment.

FIG. 6 A flowchart showing the procedure of the image processingaccording to the first exemplary embodiment.

FIG. 7 Illustrates the pixel signals of a right image capturing area inthe first exemplary embodiment.

FIG. 8 Illustrates the pixel signals of a right image capturing area inthe first exemplary embodiment (in a situation where the distance fromthe foreground subject to the background is halved).

FIG. 9 A block diagram illustrating an overall configuration for animage capture device as a second exemplary embodiment.

FIG. 10 Illustrates the rotation operation of a light-transmitting plateaccording to the second exemplary embodiment.

FIG. 11 A front view of a light-transmitting plate according to thesecond exemplary embodiment.

FIG. 12 A graph showing the transmission characteristic of a stripedcolor filter according to the second exemplary embodiment.

FIG. 13 (a) shows how the signal ΣTr changes with the horizontaldistance X in a third exemplary embodiment and (b) shows how the signalΣTr−(1/2)X changes with the horizontal distance X in the third exemplaryembodiment.

FIG. 14 Illustrates the arrangement of an image capturing systemaccording to Patent Document No. 1.

FIG. 15 Illustrates the arrangement of an image capturing systemaccording to Patent Document No. 2.

FIG. 16 Illustrates the arrangement of an image capturing systemaccording to Patent Document No. 4.

FIG. 17 Illustrates the appearance of a light beam confining plateaccording to Patent Document No. 5.

FIG. 18 Illustrates the appearance of a light beam confining plateaccording to Patent Document No. 6.

DESCRIPTION OF EMBODIMENTS

Exemplary embodiments of the present invention are outlined as follows:

(1) A 3D image capture device as an embodiment of the present inventionincludes: a light transmitting section with a transmitting area, ofwhich the spectral transmittance characteristic varies in a firstdirection; an image sensor which is arranged to receive light that hasbeen transmitted through the light transmitting section and whichoutputs a photoelectrically converted signal representing the lightreceived; an imaging section which produces an image on the imagecapturing plane of the image sensor; and an image processing sectionwhich extracts an edge of a subject in the first direction, which isincluded in an image that has been generated based on thephotoelectrically converted signal supplied from the image sensor, andwhich estimates information about the depth of the subject based onlightness or hue pattern of a background in the vicinity of the edgeextracted.

(2) In one embodiment, the transmission wavelength range of thetransmitting area changes into three or more different ones in the firstdirection.

(3) In one embodiment of the 3D image capture device of (1) or (2), thetransmitting area is designed so that when achromatic color light istransmitted through the transmitting area, the sum of transmitted lightrays becomes the achromatic color light.

(4) In one embodiment, the 3D image capture device of one of (1) to (3)includes a rotating and driving section which rotates the transmittingarea on a plane that intersects with an optical axis at right angles,and the image processing section extracts the edge of the subject bycomparing to each other a plurality of images that have been obtained inmutually different rotation states.

(5) In one embodiment of the 3D image capture device of one of (1) to(4), the image processing section extracts the edge of the subject basedon the difference between a first image obtained when the transmittingarea was in a first state and a second image obtained when thetransmitting area was in a second state, which is defined by rotatingthe transmitting area in the first state 180 degrees.

(6) In one embodiment of the 3D image capture device of one of (1) to(5), the spectral transmittance characteristic of the transmitting areachanges continuously and periodically in the first direction.

(7) In one embodiment of the 3D image capture device of one of (1) to(6), the transmitting area has six areas which are arranged in the firstdirection and which transmit light rays falling within the wavelengthranges of the colors blue, cyan, green, yellow, red and magenta,respectively.

(8) In one embodiment of the 3D image capture device of one of (1) to(7), the image processing section estimates the depth of the subject byreference to information that has been collected in advance to define arelation between the depth of the subject and a lightness or hue patternin pixels surrounding the edge.

(9) In one embodiment of the 3D image capture device of one of (1) to(8), the rest of the light transmitting section other than thetransmitting area is transparent.

(10) In one embodiment of the 3D image capture device of one of (1) to(9), the image processing section generates a depth image, of which eachpixel value is represented by the level of the depth, by reference toinformation indicating the estimated depth.

(11) In one embodiment of the 3D image capture device of one of (1) to(10), the image processing section generates a color image based on thephotoelectrically converted signal supplied from the image sensor.

(12) A light transmitting section as an embodiment of the presentinvention may be used by the 3D image capture device of one of (1) to(11).

(13) An image processor as an embodiment of the present invention may beused by the 3D image capture device of one of (1) to (11). The imageprocessor includes an image processing section which extracts an edge ofa subject in the first direction, which is included in an image that hasbeen generated based on the photoelectrically converted signal suppliedfrom the image sensor, and which estimates information about the depthof the subject based on a lightness or hue pattern of a background inthe vicinity of the edge extracted.

(14) An image processing program as an embodiment of the presentinvention may be used by the 3D image capture device of one of (1) to(11). The image processing program is defined so as to make a computerperform the steps of: extracting an edge of a subject in the firstdirection, which is included in an image that has been generated basedon the photoelectrically converted signal supplied from the imagesensor; and estimating information about the depth of the subject basedon a lightness or hue pattern of a background in the vicinity of theedge extracted.

Hereinafter, embodiments of the present invention will be described infurther detail with reference to the accompanying drawings. In thefollowing description, any element shown in multiple drawings and havingsubstantially the same function will be identified by the same referencenumeral. It should be noted that a signal or information representing animage will be sometimes referred to herein as just an “image”.

Embodiment 1

First of all, a 3D image capture device (which will be simply referredto herein as an “image capture device”) as a first embodiment of thepresent invention will be described. FIG. 1 is a block diagramillustrating an overall configuration for an image capture deviceaccording to this embodiment. The image capture device of thisembodiment is a digital electronic camera and includes an imagecapturing section 100 and a signal processing section 200 that receivesa signal generated by the image capturing section 100 and outputs asignal representing an image (i.e., an image signal).

The image capturing section 100 includes a color solid-state imagesensor 2 a (which will be simply referred to herein as an “imagesensor”) with a number of photosensitive cells (pixels) that arearranged on its image capturing plane, a light-transmitting plate(light-transmitting section) 2, in which a striped color filter, ofwhich the spectral transmittance characteristic varies in a particulardirection, is arranged and which transmits incoming light, an opticallens 3 for producing an image on the image capturing plane of the colorsolid-state image sensor 2 a, and an infrared cut filter 4. The imagecapturing section 100 further includes a signal generating and receivingsection 5, which not only generates a fundamental signal to drive thecolor solid-state image sensor 2 a but also receives the output signalof the color solid-state image sensor 2 a and sends it to the signalprocessing section 200, and a sensor driving section 6 for driving thecolor solid-state image sensor 2 a in accordance with the fundamentalsignal generated by the signal generating and receiving section 5. Thecolor solid-state image sensor 2 a is typically a CCD or CMOS sensor,which may be fabricated by known semiconductor device processingtechnologies. The signal generating and receiving section 5 and thesensor driving section 6 may be implemented as an LSI such as a CCDdriver. In this description, the “spectral transmittance characteristic”refers herein to the wavelength dependence of a transmittance in thevisible radiation wavelength range.

The signal processing section 200 includes an image processing section 7which generates a color image and subject's depth information byprocessing the signal supplied from the image capturing section 100, amemory 30 for storing various kinds of data for use to generate theimage signal, and an interface (I/F) section 8 for sending out the imagesignal and depth information thus generated to an external device. Theimage processing section 7 may be implemented appropriately as acombination of a hardware component such as a known digital signalprocessor (DSP) and a software program for use to perform imageprocessing involving the image signal generation. The image processingsection 7 not only generates a color image but also extracts an edgefrom the image and calculates depth information based on colorinformation in the vicinity of the edge. Also, the image processingsection 7 transforms the depth information into a luminance signal,thereby generating a monochrome image representing the distribution ofdepths. The memory 30 may be a DRAM, for example. And the memory 30 notonly stores the signal supplied from the image capturing section 100 butalso temporarily retains the image data that has been generated by theimage processing section 7 or compressed image data. These image dataare then output to either a storage medium or a display section (neitheris shown) by way of the interface section 8.

The image capture device of this embodiment actually further includes anelectronic shutter, a viewfinder, a power supply (or battery), aflashlight and other known components. However, the description thereofwill be omitted herein because none of them are essential componentsthat would make it difficult to understand how this embodiment worksunless they were described in detail.

Next, the configuration of the image capturing section 100 will bedescribed in further detail with reference to FIGS. 2 through 4. In thefollowing description, the x and y coordinates shown in those drawingswill be used.

FIG. 2 schematically illustrates the relative arrangement of thelight-transmitting plate 1, the lens 3 and the image sensor 2 a in theimage capturing section 100. It should be noted that illustration of theother elements is omitted in FIG. 2. The lens 3 may be a lens unit thatis a group of lenses but is drawn in FIG. 2 as a single lens for thesake of simplicity. The light-transmitting plate includes a stripedcolor filter (light transmitting area) 1 a, of which the spectraltransmittance characteristic varies in the horizontal direction, andtransmits the incoming light. The lens 3 is a known lens and condensesthe light that has been transmitted through the light-transmitting plate1, thereby imaging the light on the image capturing plane 2 a of theimage sensor 2. In this description, the “horizontal direction” meansthe x direction shown in the drawings that are referred to, and is notnecessarily the direction that is parallel to the surface of the ground.

FIG. 3 is a front view of the light-transmitting plate 1 according tothis embodiment. The light-transmitting plate 1 of this embodiment, aswell as the lens 3, has a circular shape. The light-transmitting plate 1has a striped color filter 1 a which runs through its middle but therest 1 b of the plate 1 is transparent. This striped color filter 1 a ischaracterized in that its transmission wavelength range graduallychanges in the order of the colors red (R), yellow (Ye), green (G), cyan(Cy), blue (B) and magenta (Mg) from the left to the right on thedrawing and that the sum of all of these colors becomes the color white.As a result, even though the waveform is different depending on whatportion of the striped color filter 1 a achromatic color light (i.e.,white light) has transmitted, the sum of the transmitted light becomesachromatic color light as a whole. In this embodiment, thetransmittances of the R, G and B portions of the filter are supposed tobe substantially equal to each other and the transmittances of the Ye,Cy and Mg portions of the filter are supposed to be approximately twiceas high as that of the R, G and B portions.

FIG. 4 illustrates some of a plurality of photosensitive cells 5 whichare arranged in columns and rows on the image capturing plane 2 b of theimage sensor 2 a. Each of those photosensitive cells 50 is typically aphotodiode, which performs photoelectric conversion and outputs anelectrical signal representing the quantity of the light received (whichwill be referred to herein as a “photoelectrically converted signal” ora “pixel signal”). Color filters are arranged closer to the light sourceso as to face the respective photosensitive cells 50. As shown in FIG.4, in this embodiment, the color filters have a horizontal stripedarrangement, each fundamental unit of which is comprised of three colorfilters that are arranged in three rows and one column, and red (R),green (G) and blue (B) elements are arranged on the first, second andthird rows, respectively. The color filter of each of these elements maybe made of a known pigment, for example. The color filters of the red,green and blue elements selectively transmit light rays falling withinthe red, green and blue wavelength ranges, respectively.

According to such an arrangement, the light that has entered this imagecapture device during an exposure process passes through thelight-transmitting plate 1, the lens 3, and the infrared cut filter 4,is imaged on the image capturing plane 2 b of the image sensor 2 a, andthen is photoelectrically converted by each photosensitive cells 50. Thephotoelectrically converted signal is output from each photosensitivecell 50 to the signal processing section 200 via the signal generatingand receiving section 5. In the signal processing section 200, the imageprocessing section 7 colors the image and calculates depth informationbased on the signal supplied. The depth information is transformed intoa luminance signal according to the level of its depth and is output asa monochrome image.

Next, it will be described how to perform an image capturing operationand image processing according to this embodiment.

FIG. 5 schematically illustrates a situation where images of abackground 32 and a foreground subject 31 are captured through thelight-transmitting plate 1. When their images are captured in thesituation shown in FIG. 5, the image sensor 2 a outputs aphotoelectrically converted signal based on the light that has beenreflected from the foreground subject 31 and the background 32. Thephotoelectrically converted signal is then sent from the image sensor 2a to the image processing section 7 via the signal generating andreceiving section 5. The image processing section 7 performs thefollowing two kinds of processing on the photoelectrically convertedsignal supplied, and outputs two images as the results of those twokinds of processing. Specifically, predetermined coloration processingis carried out as the first processing, thereby generating a colorimage. Next, as the second processing, an edge of the image isextracted, the depth is estimated based on the horizontal coloration inthe vicinity of the edge, and a monochrome image, of which the imageluminance value is represented by the level of the depth (which will bereferred to herein as a “depth image”), is generated. Hereinafter, itwill be described more specifically how to generate the color image andthe depth image.

FIG. 6 is a flowchart showing the procedure of the image generationprocessing according to this embodiment. First, in Step S10, the imageprocessing section 7 generates a color image based on thephotoelectrically converted signal that has been generated by the imagesensor 2 a. Next, in Step S11, the image processing section 7 extracts ahorizontal edge of the subject which is included in the color imagegenerated. Then, in Step S12, the image processing section 7 detects thecolor of the background in the vicinity of the edge. Thereafter, in StepS13, the image processing section 7 detects a hue pattern of pixels inthe vicinity of the edge. Next, in Step S14, the image processingsection 7 calculates the subject's depth based on the hue pattern ofpixels in the vicinity of the edge. Finally, in Step S15, the imageprocessing section 7 generates a depth image based on the level of thedepth calculated. Hereinafter, these processing steps will be describedin detail one by one.

First of all, in Step S10, a color image is generated based on theoutput of the image sensor 2 a. In this embodiment, the image sensor 2 ahas a horizontal stripe arrangement of RGB. That is why RGB signals(i.e., color signals) can be obtained directly by using three pixelsthat are arranged in three rows and one column as a unit, and colorimage is generated based on these color signals. Particularly, since thehorizontal stripe color arrangement shown in FIG. 4 is used according tothis embodiment, a color image with high horizontal resolution can begenerated. Since the striped color filter 1 a is arranged in thisembodiment to run through the middle of the light-transmitting plate 1,light is partially absorbed into the color filter 1 a and some quantityof the light is lost. However, the rest of the light is not lost but isphotoelectrically converted. In addition, according to this embodiment,the color scheme of the striped color filter 1 a includes not onlyprimary colors (R, G, B) but also complementary colors (Ye, Cy, Mg) aswell. That is why higher optical transmittance is achieved than in anarrangement that uses only primary color filters. On top of that, sincethe sum of the light rays that are transmitted through the striped colorfilter 1 a is white light, basically the subject image is not colored bythe striped color filter 1 a. Thus, it can be seen that a color image,of which the sensitivity and color properties have no problem at all,can be generated by setting the size of the striped color filter 1 a tobe sufficiently small.

Next, in Step S11, the image processing section 7 extracts a horizontaledge of the image that has been obtained by capturing (i.e., the imagecaptured). The edge of the image can be extracted by several methods,any of which may be adopted. In this embodiment, the edge is supposed tobe extracted based on the result of the color image generationprocessing described above. Specifically, first of all, color componentsare removed from the color image that has been obtained through thecolor image generation processing described above, thereby obtaining amonochrome image. Next, a signal difference is calculated between twohorizontally adjacent pixels of the monochrome image. And if thedifferential value is equal to or greater than a preset level, then thatpart is regarded as the horizontal edge of the image. By performing suchsignal difference processing on the entire image, the image processingsection 7 extracts horizontal edges.

Subsequently, in Step S12, the color of the background in the vicinityof the edge extracted is detected. The color is detected from an areathat is located on the left-hand side of the edge (e.g., correspondingto the left image capturing area 33 shown in FIG. 5) in the left half ofthe image and from an area that is located on the right-hand side of theedge (e.g., corresponding to the right image capturing area 34 shown inFIG. 5) in the right half of the image. It should be noted that thespecific area widths (i.e., the numbers of horizontal pixels) of theleft and right areas 33 and 34 are set so as to correspond to theexpected depth range of the image capturing area.

For example, suppose the color of the right image capturing area shownin FIG. 5 is white and the area ratio of the striped color filter 1 a ofthe light-transmitting plate 1 shown in FIG. 3 to the other transparentarea 1 b is 1:k. In that case, the image areas corresponding to the leftand right image capturing areas 34 and 34 are affected by the stripedcolor filter 1 a and their colors change continuously in the horizontaldirection. For instance, a light ray that has been incident on the imagesensor 2 a from the leftmost portion of the right image capturing area34 shown in FIG. 5 transmits only through the right end (Mg area) of thestriped color filter 1 a, and therefore, pixels corresponding to thatportion are colored Mg. On the other hand, since a light ray that hascome from the rightmost portion of the right image capturing area 34transmits only through the entire striped color filter 1 a and thenimpinges on the image sensor 2 a, pixels corresponding to that portionare not colored.

FIG. 7 schematically illustrates the signal levels of six horizontalpixels on the image that correspond to the right image capturing area34. As shown in FIG. 7, the signal levels increase from a pixelcorresponding to the leftmost portion of the right image capturing area34 toward a pixel corresponding to the rightmost portion thereof. InFIG. 7, signals representing seven pixels that are arranged horizontallyare identified by S(i), S(i+1), S(i+2) . . . and S(i+6) and an edge issupposed to be present in the boundary between S(i) and S(i+1). Also, W(==R+G+B) component of each pixel signal is identified by k(j) and j==1to 6 and k(6)==k are supposed to be satisfied. If a signal indicatingthe intensity of each color component included in one of these signalsis represented by the sign indicating that color component (Mg, B, Cy,G, Ye, R), a pixel signal S(i+1) corresponding to the leftmost portionof the right image capturing area 34 is represented by Mg+k(1) (R+G+B).A pixel signal S(i+6) corresponding to the rightmost portion of theright image capturing area 34 is represented by(Mg+B+Cy+G+Ye+R)+k(6)(R+G+B). In the same way, as for the other pixelsignals, S(i+2) is represented by (Mg+B)+k(2) (R+G+B), S(i+3) isrepresented by (Mg+B+Cy)+k(3) (R+G+B), S(i+4) is represented by(Mg+B+Cy+G)+k(4) (R+G+B), and S(i+5) is represented by(Mg+B+Cy+G+Ye)+k(5) (R+G+B). In this case, however, the pixel unit issupposed to be comprised of the three pixels that are arranged in threerows and one column shown in FIG. 4 and those three pixels are supposedto form a single pixel.

After all, in the entire right image capturing area 34, the sum of thepixel signals is represented by 9R+9G+15B+Σk(j) (R+G+B) and the RGBsignal ratio becomes R:G:B=(9+Σk(j)):(9+Σk(j)):(15+Σk(j)). As k(j) is aknown number determined by the lens shape, this RGB ratio can beobtained. As the RGB ratio varies according to the color of thebackground in the vicinity of an edge, the color of the background canbe determined by obtaining the RGB ratio. For example, if the RGB ratiois close to R:G:B=0:(9+Σk(j)):(15+Σk(j)), the background color can bedetermined to be Cy. In this embodiment, a database that associates anRGB ratio with its corresponding background color is collected inadvance and stored in the memory 30. By reference to that database, theimage processing section 7 detects the color of the background area inthe vicinity of an edge based on the RGB ratio of pixels in the vicinityof the edge on the image.

In Step S13, the hue pattern in an area surrounding the edge isdetected. Specifically, signals representing the color components thathave been detected in Step S12 are removed from respective pixel signalsin the edge surrounding area and then the difference is calculatedbetween the pixels, thereby calculating a color component. For instance,since the color of the right image capturing area 34 is white in theexample shown in FIG. 7, a white component k(j)(R+G+B) is removed fromthe respective pixel signals S(i+1), S(i+2) and so on. Then, the pixelsignals S(i+1) to S(i+6) in the right image capturing area 34 arerepresented by the following Equations (1) through (6):S(i+1)=Mg  (1)S(i+2)=Mg+B  (2)S(i+3)=Mg+B±Cy  (3)S(i+4)=Mg+B±Cy+G  (4)S(i+5)=Mg+B±Cy+G+Ye  (5)S(i+6)=Mg+B±Cy+G+Ye+R  (6)

By further calculating the differences between the respective pixelsignals, the following Equations (7) through (11) can be obtained. InEquations (7) through (11), the pixel difference signal Djk is supposedto be calculated by Djk=S(i+k)−S(i+j).D12=S(i+2)−S(i+1)=B  (7)D23=S(i+3)−S(i+2)=Cy  (8)D34=S(i+4)−S(i+3)=G  (9)D45=S(i+5)−S(i+4)=Ye  (10)D56=S(i+6)−S(i+5)=R  (11)

As a result of these computations, by combining D12 through D56 andS(i+1), signals Mg through R representing the colors of the stripedcolor filter 1 a when the light source is white can be calculated. Itcan be said that this is equivalent to calculating the colors to betransmitted through the striped color filter using the color of the edgesurrounding areas such as the right and left image areas 34 and 33 asthe color of a light source.

In Step S14, a property of the color pattern surrounding the edge isexamined and the depth corresponding to that property is calculated. Inthis description, the depth refers herein to the distance from thesubject 31 to the background 32 shown in FIG. 5. If the depth/distanceis short, then the hue pattern is limited. Conversely, if thedepth/distance is long, then the hue pattern range expands. That is tosay, there is a correlation between the subject's (31) depth informationand the hue pattern in the vicinity of the edge. In this embodiment,data representing that correlation is provided in advance and stored inthe memory 30. By reference to that correlation data, the imageprocessing section 7 calculates the depth based on the hue pattern atthe edge.

For example, if in a situation where the hue pattern in the right imagecapturing area 34 is represented by Equation (1) and Equations (7)through (11) (i.e., in a situation where six colors can be obtained fromsix pixels) the distance from the foreground subject 31 to thebackground 32 becomes a half, then three colors can be obtained fromthree pixels as shown in FIG. 8. Thus, it can be seen that there is acorrelation between the hue pattern obtained and the distance from theforeground subject 31 to the background 32. The image processing section7 of this embodiment obtains depth information based on thatcorrelation. Nevertheless, the depth information is not absolutedistance information as in the example described above but just a pieceof relative information.

Next, in Step S15, the depth information that has been obtained in StepS14 is transformed into brightness information (i.e., a monochromeimage). Specifically, the image processing section 7 scans the capturedimage horizontally, and calculates the depth every time an edge isdetected. Then, pieces of depth information calculated are accumulatedand added together, thereby calculating a maximum depth value. By usingthe maximum depth value as a maximum brightness value, the brightness ofeach pixel is determined. For example, in transforming the depthinformation into an eight-bit monochrome image, the information can betransformed into an image with a maximum luminance value of 255.

As can be seen from the foregoing description, by arranging thelight-transmitting plate 1 with the striped color filter 1 a in thecolor image capturing system, the image capture device of thisembodiment can obtain not only a color image with only a little loss oflight but also the subject's depth image. According to this embodiment,by extracting an edge from the color image obtained and by examining thehue of the colors in the area surrounding that edge, a relative depthbetween the subjects can be calculated, which is very advantageous. Ontop of that, as information about, the relative depth between thesubjects can be obtained, information about the subject's depth withrespect to the image capture device's position can also be obtained bymaking computations based on that relative depth information.

In the foregoing description, each filter portion of the striped colorfilter 1 a is supposed to be configured to transmit only a light rayfalling within its associated particular wavelength range but not totransmit any other light ray. Also, the R, G and B filter portions aresupposed to have substantially the same transmittance and thetransmittance of the Ye, Cy and Mg filter portions is supposed to beapproximately twice as high as that of the R, G and B filter portions.However, the striped color filter 1 a does not have to satisfy theseconditions exactly. Naturally, these conditions are ideally satisfiedbut even if the characteristic of the striped color filter 1 a variedfrom the ideal one, there would be no problem if the signal processingdescribed above is corrected so as to compensate for that variation.

In the embodiment described above, the color scheme of the striped colorfilter 1 a of the light-transmitting plate 1 is supposed to be comprisedof red (R), yellow (Ye), green (G), cyan (Cy), blue (B) and magenta(Mg). However, this is only an example and any other color scheme may beadopted as well. Furthermore, the number of different color schemes ofthe striped color filter 1 a does not have to be seven. Nevertheless, inorder to increase the accuracy of calculating the depth, it isrecommended that there be at least three different color schemes. Also,in order to obtain an image that can be used as a color image with noproblem, the sum of all colors of the striped color filter 1 a suitablybecomes the color white as in the embodiment described above. But thesum does not have to be exactly the color white. However, if a stripedcolor filter 1 a, of which the sum of all colors is close to the colorwhite, is used, the image obtained can be used as a color image with noproblem at all and yet the depth information can also be calculated.

Furthermore, in the embodiment described above, the basic colorarrangement of the color image sensor 2 a is supposed to be an RGBhorizontal stripe arrangement. However, this is only an example.Alternatively, any other basic color arrangement consisting of three ormore colors may also be used. For example, even though the depth cannotbe calculated as accurately as with the horizontal stripe arrangement, aBayer arrangement consisting of red, blue and two green elements mayalso be used with no problem at all. Also, although the condenser lens 3and the light-transmitting plate 1 are supposed to be circular, theeffects will not be affected at all even if the condenser lens 3 and thelight-transmitting plate 1 have a quadrangle or any other arbitraryshape. Furthermore, even though the color filter 1 a of thelight-transmitting plate 1 is supposed to have a striped shape, this isonly an example and the color filter may also cover the entire surfaceof the light-transmitting plate 1. For example, if a filter, of whichthe hue (i.e., transmission wavelength range) changes in the rotatingdirection and the color depth (i.e., transmittance) changes in theradial direction as in a Munsell color ring, is used, then depth can becalculated not just in the horizontal direction but also in every otherdirection as well. Still alternatively, a filter, of which thetransmission wavelength range or transmittance changes concentrically,may also be used. In any of these filters, the transmittancecharacteristic is suitably designed so that the sum of the transmittedlight rays becomes close to white light.

It should be noted that the arrangement of the respective members shownin FIG. 2 is only an example. And the present invention is in no waylimited to that specific example. Alternatively, as long as an image canbe produced on the image capturing plane 2 b, the lens 3 may be arrangedmore distant from the image sensor 2 a than the light-transmitting plate1 is. Still alternatively, the lens 3 and the light-transmitting plate 1may also be implemented as a single optical element.

In the embodiment described above, the image processing section 7generates a color image and a depth image at the same time. However, theimage processing section 7 may generate only a depth image withoutgenerating any color image. Alternatively, the image processing sectionmay also be configured to generate only depth information, not any depthimage, by performing the processing described above. Furthermore, theimage processing of this embodiment may also be carried out by anotherdevice that is provided independently of that image capture device. Forexample, even if a signal that has been obtained by an image capturedevice including the image capturing section 100 of this embodiment isloaded into another device (image processor) to get a program definingthe signal arithmetic processing described above executed by a computerbuilt in that image processor, the effects of the embodiment describedabove can also be achieved.

Embodiment 2

Hereinafter, a second embodiment of the present invention will bedescribed. In this embodiment, a rotating mechanism is attached to thelight-transmitting plate 1 and rotates the light-transmitting plate 1 ona plane that intersects with the optical axis at right angles, therebygetting images captured twice in a row. In the other respects, however,the configuration of this second embodiment is the same as that of thefirst embodiment, and their common features will not be described allover again to avoid redundancies.

FIG. 9 is a block diagram illustrating an overall configuration for animage capture device according to this embodiment. The image capturedevice of this embodiment includes a rotating and driving section 40 asa rotating mechanism. The rotating and driving section 40 has a motorthat rotates the light-transmitting plate 1 and turns thelight-transmitting plate 1 in accordance with an instruction given bythe sensor driving section 6. Specifically, as disclosed in Non-PatentDocument No. 1, the light-transmitting plate 1 may be rotated by puttinga belt on the light-transmitting plate 1 and by running the belt with amotor.

Hereinafter, it will be described how to perform an image capturingoperation according to this embodiment. shown in FIG. 10, the imagecapture device captures an image once with the striped color filter 1 aof the light-transmitting plate 1 kept parallel to the horizontaldirection (see portion (a)), turns the light-transmitting plate 1 180degrees, and captures an image once again (see portion (b)). These twoimages captured are subjected to differential processing and differentportions as image data are extracted based on the result of thedifferential processing. In this manner, areas on the imagecorresponding to the left and right image capturing areas 33 and 34 canbe located

Even though the left and right image capturing areas 33 and 34 cannot belocated exactly according to the first embodiment described above, thoseareas can be located accurately according to this embodiment. The reasonis that the pixels that define the left and right image capturing areas33 and 34 come to have different signal levels as the color scheme ofthe striped color filter 1 a changes every time an image is captured.That is why based on the difference between these two images, the leftand right image capturing areas 33 and 34 can be located accurately.Once the left and right image capturing areas 33 and 34 have beenlocated, the same processing is carried out as in the first embodimentdescribed above. According to this embodiment, since the left and rightimage capturing areas 33 and 34 can be located exactly, the depth can becalculated more accurately.

As can be seen, according to this embodiment, by rotating thelight-transmitting plate 1 180 degrees and capturing images twice beforeand after the rotation, the left and right image capturing areas 33 and34 can be located accurately based on the difference between the twoimages captured. Consequently, the depth can be calculated even moreaccurately.

Even though images are supposed to be captured twice according to thisembodiment by getting the light-transmitting plate 1 rotated 180 degreesby the rotating and driving section 40, the angle of rotation and thenumber of times of capturing images may be changed. For example, twopairs of multi-viewpoint images, which are located at the upper, lower,right and left corners, respectively, may also be generated byperforming image capturing operations four times with thelight-transmitting plate 1 rotated 90 degrees each time.

Embodiment 3

Hereinafter, a third embodiment of the present invention will bedescribed. In this embodiment, the striped color filter 1 a of thelight-transmitting plate 1 is arranged as in the first embodimentdescribed above but its spectral transmittance characteristic changesdifferently from in the first embodiment. Specifically, the stripedcolor filter 1 a of this embodiment has the color grey (i.e., has nowavelength selectivity) and its transmittance changes periodically inthe horizontal direction (i.e., in the x direction). Since the colorfilter 1 a is a grey filter, the depth is calculated according to thisembodiment based on the luminance signal of a color image, not RGB pixelsignals. Thus, the following description of the third embodiment will befocused on only those differences from the first embodiment and theircommon features will not be described all over again to avoidredundancies.

FIG. 11 is a front view of the light-transmitting plate 1 according tothis embodiment. In this embodiment, the striped color filter 1 a doesnot have wavelength selectivity but its transmittance changes just likea cos function and becomes outstandingly high at two points and very lowat two points. FIG. 12 is a graph showing how its transmittance changes.In FIG. 12, the abscissa represents a coordinate x on the color filter 1a and the ordinate represents a transmittance Tr, which can becalculated by the following Equation (12):Tr==(½)cos X+½  (12)

If a striped color filter 1 a with such a transmittance distribution isused, the luminance values of image areas corresponding to the left andright image capturing areas 33 and 34 are obtained by finding anintegral of the transmittance of the striped color filter 1 ahorizontally. In this embodiment, since the optical transmittance Tr isa periodic function represented by Equation (12), its integral value ΣTrcan be represented as the sum of a term of the periodic function and aterm of the linear function as in the following Equation (13):ΣTr=(1/2)sin X+(1/2)X  (13)

FIG. 13(a) is a graph showing the integral value E Tr given by Equation(13). The periodic function of the first term before and after theintegral operation is characterized by its waveform that does not changebecause the cos function just turns into a sin function. Thus, the imageprocessing section 7 of this embodiment removes a signal correspondingto the linear function term of Equation (13) from pixel signalscorresponding to the left and right image capturing areas 33 and 34 andthen analyzes the waveforms of signals of multiple pixels that arearranged horizontally. FIG. 13(b) is a graph showing a function that isobtained by removing a linear function term from Tr represented byEquation (13). The image processing section 7 detects a similar waveformto that of the periodic function shown in FIG. 13(b) from the pixelsignals of multiple pixels corresponding to the left and right imagecapturing areas 33 and 34 and analyzes that waveform, thereby measuringthe depth. Specifically, information representing the relation betweenthe depth and the waveform of the periodic function (e.g., wavelength)is collected in advance and stored in the memory 30. By reference tothat information, the image processing section 7 obtains the depth basedon the waveform of the periodic function that is based on the imagedata.

As described above, according to this embodiment, a light-transmittingplate 1, of which the striped color filter 1 a has the color grey and ofwhich the transmittance changes periodically in the horizontaldirection, is used. With such a configuration adopted, the waveform of aperiodic function appears in the luminance values of image areascorresponding to the left and right image capturing areas 33 and 34 andthe depth can be calculated based on that waveform. Consequently,according to this embodiment, information about the depth of the subjectcan be estimated based on a lightness pattern of the background in thevicinity of an edge.

In the embodiment described above, the striped color filter 1 a of thelight-transmitting plate is supposed to have the color grey. However,this is only an example. The striped color filter 1 a may have any othercolor as long as its transmittance changed periodically. Also, as thedepth information is estimated based on the luminance signals ofrespective pixels, the striped color filter 1 a may also have wavelengthselectivity. Even if the spectral transmittance characteristic patternof the striped color filter 1 a is different from what has already beendescribed, the depth information can also be obtained by performingsignal processing according to the pattern of its spectral transmittancecharacteristic.

Also, since luminance signals are used to calculate the depthinformation in the embodiment described above, the image sensor 2 a doesnot have to be a color image sensor but may also be a monochrome imagesensor as well. Furthermore, the edge may also be extracted according tothis embodiment by rotating the light-transmitting plate 1 as in thesecond embodiment described above.

INDUSTRIAL APPLICABILITY

A 3D image capture device according to an embodiment of the presentinvention can be used effectively in any camera that ever uses asolid-state image sensor. Examples of those cameras include consumerelectronic cameras such as digital cameras and digital camcorders andsolid-state surveillance cameras for industrial use.

REFERENCE SIGNS LIST

-   1 light-transmitting plate-   1 a striped color filter-   1 b transparent area-   2 solid-state image sensor-   2 a color solid-state image sensor-   2 b image capturing plane-   3 optical lens-   4 infrared cut filter-   5 signal generating and receiving section-   6 sensor driving section-   7 image processing section-   8 image interface section-   9 image capture device-   10 pixel-   11 0-degree-polarization polarizer-   12 90-degree-polarization polarizer-   13 reflective mirror-   14 half mirror-   15 circular polarization filter-   16 driver that rotates polarization filter-   17, 18 polarization filter-   19 lens diaphragm-   20, 22, 23 light beam confining plate-   20 a color filter that transmits red-based ray-   20 b color filter that transmits blue-based ray-   21 photosensitive film-   22R, 23R R ray transmitting area of light beam confining plate-   22G, 23G G ray transmitting area of light beam confining plate-   22B, 23B B ray transmitting area of light beam confining plate-   30 memory-   31 foreground subject-   32 background-   33 left image capturing area-   34 right image capturing area-   40 rotating and driving section-   50 pixel

The invention claimed is:
 1. A 3D image capture device comprising: alight transmitting section with a transmitting area, wherein thetransmitting area has at least three areas which are arranged in thefirst direction and which transmit light rays falling within respectivedifferent wavelength ranges; an image sensor which is arranged toreceive light that has been transmitted through the light transmittingsection and which outputs a photoelectrically converted signalrepresenting the light received; an optical lens which produces an imageon the image capturing plane of the image sensor; and an image processorconfigured to: receive the image that has been generated based on thephotoelectrically converted signal supplied from the image sensor;identify a subject in the image; identify an edge of the subject,wherein the edge lies along the first direction; identify a backgroundin the vicinity of the edge and separate from the subject; determine amagnitude of variation in the first direction of a lightness or hue ofthe background, wherein the determined magnitude of variationcorresponds to a number of levels having a different intensity or anumber of different colors of the background, respectively; estimate adistance between the subject and the background based on the determinedmagnitude of variation and a predetermined correlation between themagnitude of variation and distance, wherein a larger determinedmagnitude of variation corresponds to a larger estimated distance. 2.The 3D image capture device of claim 1, wherein the transmissionwavelength range of the transmitting area changes into three or moredifferent ones in the first direction.
 3. The 3D image capture device ofclaim 1, wherein the transmitting area is designed so that whenachromatic color light is transmitted through the transmitting area, thesum of transmitted light rays becomes the achromatic color light.
 4. The3D image capture device of claim 1, comprising a motor which rotates thetransmitting area on a plane that intersects with an optical axis atright angles, wherein the image processor identifies the edge of thesubject by comparing to each other a plurality of images that have beenobtained in mutually different rotation states.
 5. The 3D image capturedevice of claim 4, wherein the image processor identifies the edge ofthe subject based on the difference between a first image obtained whenthe transmitting area was in a first state and a second image obtainedwhen the transmitting area was in a second state, which is defined byrotating the transmitting area in the first state 180 degrees.
 6. The 3Dimage capture device of claim 1, wherein the spectral transmittancecharacteristic of the transmitting area changes continuously andperiodically in the first direction.
 7. The 3D image capture device ofclaim 1, wherein the transmitting area has six areas which are arrangedin the first direction and which transmit light rays falling within thewavelength ranges of the colors blue, cyan, green, yellow, red andmagenta, respectively.
 8. The 3D image capture device of claim 1,wherein the rest of the light transmitting section other than thetransmitting area is transparent.
 9. The 3D image capture device ofclaim 1, wherein the image processor generates a depth image, of whicheach pixel value is represented by the level of the depth, by referenceto information indicating the distance between the subject and thebackground.
 10. The 3D image capture device of claim 1, wherein theimage processor generates a color image based on the photoelectricallyconverted signal supplied from the image sensor.
 11. A lighttransmitting section to be used by the 3D image capture device ofclaim
 1. 12. A processor to be used by the 3D image capture device ofclaim 1, the processor comprising the image processor of claim
 1. 13. Acomputer program, stored on a non-transitory computer-readable medium,to be executed by a computer mounted in the 3D image capture device ofclaim 1, the computer program causing the computer to execute the stepsof: identifying a subject in a received image; identifying an edge ofthe subject; identifying an edge of the subject, wherein the edge liesalong the first direction identifying a background in the vicinity ofthe edge and separate from the subject; determining a magnitude ofvariation in the first direction of a lightness or hue of thebackground, wherein the determined magnitude of variation corresponds toa number of levels having a different intensity or a number of differentcolors of the background, respectively; estimating a distance betweenthe subject and the background based on the determined magnitude ofvariation and a predetermined correlation between the magnitude ofvariation and distance, wherein a larger determined magnitude ofvariation corresponds to a larger estimated distance.